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A METHOD AND SYSTEM FOR CONTROLLING USER INTERACTION 
IN A DIGITAL IMAGING DEVICE USING DYNAMIC OVERLAY BARS 

CROSS-REFERENCE TO RELATED APPLICATIONS 

5 This application is related to U.S. Patent Application No. 08/939,993 filed on 

September 26, 1997, entitled "A Method And System For Manipulating Images Stored 

In A Digital Imaging Device" (P110CIP); and U.S. Patent Application No. , entitled 

"Directing Image Capture Sequences In A Digital Imaging Device Using Scripts" (P165), 
and U.S. Patent Application No. , entitled "Method and System For Displaying 

1 0 Overlay Bars In A Digital Imaging Device" (P1 66), filed on the same date as the present 
application. 



FIELD OF THE INVENTION 

fU The present invention relates generally to digital imaging devices, including 

Q 15 digital cameras, and more particularly to a method and system for controlling user 
interaction in a digital imaging device using dynamic overlay bars. 

BACKGROUND OF THE INVENTION 

Most digital cameras today are similar in size to and behave like conventional 
20 point-and-shoot cameras. Unlike conventional cameras, however, most digital cameras 
store digital images in an internal flash memory or on external memory cards, and some 
are equipped with a liquid-crystal display (LCD) screen on the back of the camera. 
Through the use of the LCD, most digital cameras operate in two modes, record and 
play, although some only have a record mode. 
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In record mode, which is also referred to as capture mode, the LCD acts as a live 
viewfinder in which the user may view an object or scene before taking a picture, similar 
to the LCD on a camcorder. When the user presses the shutter button, whatever scene 
is shown on the LCD is captured as a still image. Besides capturing still images, some 
5 digital cameras can be set to capture other image types, such as burst and time-lapse 
images. A burst image is a series of still images captured in rapid succession, while a 
time-lapse image is series of still images taken at regular intervals over a longer time 
period. 

In play mode, the LCD acts as a playback screen for reviewing the previously 
1 0 captured images. Typically, several small images are displayed on the LCD at once, 
Q and by selecting one of the images the user may then display the full-sized version of 
M- the images in the LCD. 

fu Although conventional digital cameras are more convenient for the user to use 

2 than film cameras due to instant play back of captured images, there are several 
y\ 15 drawbacks in the user interface that restrict user interaction with the camera. When 
2r capturing images, for example, it is often helpful for the user to be informed about the 
current settings or operational state of the camera, such as whether the flash is on/off, 
and the current image type setting, for instance. 

In conventional digital cameras, such status information is typically displayed as 
20 text blocks or accessed through a status screen or the like. The disadvantage with the 
text blocks is that they are typically small (10-15 characters in length), and therefore, 
the amount of status information they can provide is very limited. Typically, text blocks 
are used to display information such as the current image number. Moreover, when 



P135 



-2- 



text blocks are displayed with a solid color background, the background obscures that 
portion of the image. And when text blocks are displayed with no background (only 
text), the text is difficult to distinguish from the colors comprising the image, making the 
text hard to read. 

5 The disadvantage with status screens is that in order to view the status 

information, the image currently displayed on the LCD must be replaced with the status 
screen, causing the user to loose sight of the image. Another approach would be to 
shrink the display area of the LCD and add a black status area in the viewfinder, as 
done in optical viewfinders of film cameras. This, however, would shrink the size of 

1 0 images displayed in the viewfinder. 

Another drawback with conventional digital cameras is that as technological 
advances are made, digital cameras are continually provided with more features and 
functions, which make them more complex for the user to interact with. This is similar 
to what occurs with PC software, which increasingly grows larger and harder to use. 

15 PC developers attempt to alleviate this problem by providing more and larger help 
menus. Each help menu usually opens in its own window with paragraphs of scrolling 
text. 

Using PC help menus in a digital camera to guide user interaction through the 
camera features and functions would be less than ideal because of the limited size of 
20 the camera LCD. And assuming help menus were displayed, they would either obscure 
whatever image was being displayed or otherwise total replace it, which is 
disadvantageous to the picture taker. 
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Accordingly, what is needed is an improved system and method for displaying 
status information in a manner that does not obscure the display of the current object in 
the LCD, and for controlling user interaction in a digital imaging device. The present 
invention addresses such a need. 

5 

SUMMARY OF THE INVENTION 

The present invention provides a method and system for controlling user 
interaction in a digital imaging device having a display using dynamic overlay bars. The 
digital imaging device includes at least two operating modes, where each of the 
10 operating modes has at least one mode-specific operation that can be performed on 
images. In response to operating in either of the operating modes, the digital imaging 
device displays a translucent overlay bar on the display that is dynamically updated with 
status information and interactive instructions that guide the user through the mode- 
specific operations. 

15 In a second aspect of the present invention, the interactive instructions are 

implemented using a script, which is a text-based program that may be easily written by 
the user and externally loaded into the camera. Once loaded into the camera, the 
commands comprising the script are translated and executed one-by-one by a script 
interpreter to guide the user through the newly provided function. 

20 A third aspect of the present invention, provides a method and system for 

displaying overlay bars on the display. First, text and graphic information to be 
displayed on the overlay bars are stored in an overlay bar buffer, and then displayed on 
the display. Thereafter, the current image is displayed on the display line-by-line. The 
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lines of the image that will be displayed within the area of an overlay bar are stored in a 
backstore buffer. Each line in the backstore buffer is merged with its corresponding 
lines in the overlay bar buffer and displayed. This aspect of the present invention 
makes the overlay bars appear translucent, and the image appear as though it is sliding 
5 beneath the overlay bars as it is being displayed. When the user turns-off the overlay 
bars, only the portions of the image stored in the backstore buffer need be re-displayed 
to provide the original image, thus eliminating the need to re-display the entire image. 

Accordingly, the method and system of the present invention provides status 
information to a user and allows the user to perform complex camera functions and 

10 features to the images with minimum effort, while allowing for easy viewing of the 
images. Displaying interactive instructions on dynamic overlay bars to guide the user 
through complex tasks in accordance with the present invention eliminates the need for 
help screens and for the user to remember complicated key sequences, and increases 
the ease of use and operation of the digital camera. The manner in which the overlay 

15 bars and the image is displayed makes the user interface more aesthetically pleasing, 
while increasing the display speed of the digital imaging device. 

BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 is a block diagram of a digital camera that operates in accordance with 
20 the present invention. 

FIG. 2 is a block diagram of an example embodiment for the imaging device of 

FIG. 1. 

FIG. 3 is a block diagram of an example embodiment for the computer of FIG. 1 . 
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FIGS. 4 and 5 are diagrams depicting the preferred embodiment of the camera's 
1 1 0 user interface. 

FIG. 6 is a flow chart is shown illustrating the process of controlling user 
interaction in a digital imaging device using dynamic overlay bars in accordance with 
5 the present invention 

FIGS. 7A and 7B are diagrams illustrating the use of dynamic overlay bars on 
the LCD screen during capture and play modes, respectively. 

FIGS. 8A through 8C are diagrams illustrating how the overlay bars may be used 
to guide the user through a recording of a sound annotation. 
JSMO FIGS. 9A and 9B are diagrams illustrating example directed image capture 

HI screens, 

M FIG. 10 is a block diagram illustrating the camera software, which is stored in 

III ROM, and DRAM, where the software is executed. 

Q FIG. 1 1 is a flow chart illustrating an exemplary process of installing and running 

J 1 ; 15 a script-based directed image capture in a preferred embodiment of the present 
2} invention. 

FIG. 12A is a diagram illustrating a memory buffer organization for displaying 
overlay bars. 

FIG. 12B is a flow chart illustrating the process of displaying overlay bars on the 
20 LCD in accordance with the present invention. 
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DETAILED DESCRIPTION OF THE INVENTION 

The present invention relates to an improved method and system for controlling 
user interaction in a digital imaging device using dynamic overlay bars. The following 
description is presented to enable one of ordinary skill in the art to make and use the 
invention and is provided in the context of a patent application and its requirements. 
Although the present invention will be described in the context of a digital camera, 
various modifications to the preferred embodiment will be readily apparent to those 
skilled in the art and the generic principles herein may be applied to other 
embodiments. That is, any digital imaging device which displays images, icons and/or 
other items, could incorporate the features described herein below and that device 
would be within the spirit and scope of the present invention. Thus, the present 
invention is not intended to be limited to the embodiment shown but is to be accorded 
the widest scope consistent with the principles and features described herein. 

The present invention is a method and system for controlling user interaction in a 
digital imaging device using dynamic overlay bars. According to the present invention, 
both status information and interactive instructions are displayed on dynamic overlay 
bars to enable a user to perform complex camera functions and apply features to the 
images with minimum effort, while allowing for easy viewing of the images. 

Referring now to FIG. 1, a block diagram of a digital camera 110 is shown for 
use in accordance with the present invention. Camera 110 preferably comprises an 
imaging device 114, a system bus 116 and a computer 118. Imaging device 114 is 
optically coupled to an object 112 and electrically coupled via system bus 116 to 
computer 118. Once a photographer has focused imaging device 114 on object 112 
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and, using a capture button or some other means, instructed camera 1 10 to capture an 
image of object 1 12, computer 118 commands imaging device 1 14 via system bus 116 
to capture raw image data representing object 112. The captured raw image data is 
transferred over system bus 116 to computer 118 which performs various image 
5 processing functions on the image data before storing it in its internal memory. System 
bus 116 also passes various status and control signals between imaging device 114 
and computer 118. 

Referring now to FIG. 2, a block diagram of an example embodiment of imaging 
device 1 14 is shown. Imaging device 114 typically comprises a lens 220 having an iris, 
^ 10 a filter 222, an image sensor 224, a timing generator 226, an analog signal processor 
HI (ASP) 228, an analog-to-digital (A/D) converter 230, an interface 232, and one or more 
§i motors 234. 

f!j In operation, imaging device 114 captures an image of object 112 via reflected 

Q light impacting image sensor 224 along optical path 236. Image sensor 224, which is 
]}\ 15 typically a charged coupled device (CCD), responsively generates a set of raw image 
?=■ data in CCD format representing the captured image 112. The raw image data is then 
routed through ASP 228, A/D converter 230 and interface 232. Interface 232 has 
outputs for controlling ASP 228, motors 234 and timing generator 226. From interface 
232, the raw image data passes over system bus 1 16 to computer 118. 
20 Referring now to FIG. 3, a block diagram of an example embodiment for 

computer 118 is shown. System bus 116 provides connection paths between imaging 
device 114, an optional power manager 342, central processing unit (CPU) 344, 
dynamic random-access memory (DRAM) 346, input/output interface (I/O) 348, non- 



P135 



volatile memory 350, and buffers/connector 352. Removable memory 354 connects to 
system bus 116 via buffers/connector 352. Alternately, camera 110 may be 
implemented without removable memory 354 or buffers/connector 352. 

Power manager 342 communicates via line 366 with power supply 356 and 
5 coordinates power management operations for camera 110. CPU 344 typically 
includes a conventional processor device for controlling the operation of camera 1 1 0. 
In the preferred embodiment, CPU 344 is capable of concurrently running multiple 
software routines to control the various processes of camera 110 within a multithreaded 
environment. DRAM 346 is a contiguous block of dynamic memory which may be 

10 selectively allocated to various storage functions. LCD controller 390 accesses DRAM 
346 and transfers processed image data to LCD screen 402 for display. 

I/O 348 is an interface device allowing communications to and from computer 
118. For example, I/O 348 permits an external host computer (not shown) to connect to 
and communicate with computer 118. I/O 348 also interfaces with a plurality of buttons 

15 and/or dials 404, and an optional status LCD 406, which in addition to the LCD screen 
402, are the hardware elements of the camera's user interface 408. 

Non-volatile memory 350, which may typically comprise a conventional read-only 
memory or flash memory, stores a set of computer-readable program instructions to 
control the operation of camera 110. Removable memory 354 serves as an additional 

20 image data storage area and is preferably a non-volatile device, readily removable and 
replaceable by a camera 110 user via buffers/connector 352. Thus, a user who 
possesses several removable memories 354 may replace a full removable memory 354 
with an empty removable memory 354 to effectively expand the picture-taking capacity 
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of camera 110. In the preferred embodiment of the present invention, removable 
memory 354 is typically implemented using a flash disk. 

Power supply 356 supplies operating power to the various components of 
camera 110. In the preferred embodiment, power supply 356 provides operating power 
5 to a main power bus 362 and also to a secondary power bus 364. The main power bus 
362 provides power to imaging device 114, I/O 348, non-volatile memory 350 and 
removable memory 354. The secondary power bus 364 provides power to power 
manager 342, CPU 344 and DRAM 346. 

Power supply 356 is connected to main batteries 358 and also to backup 

10 batteries 360. In the preferred embodiment, a camera 110 user may also connect 
power supply 356 to an external power source. During normal operation of power 
supply 356, the main batteries 358 provide operating power to power supply 356 which 
then provides the operating power to camera 110 via both main power bus 362 and 
secondary power bus 364. During a power failure mode in which the main batteries 

15 358 have failed (when their output voltage has fallen below a minimum operational 
voltage level) the backup batteries 360 provide operating power to power supply 356 
which then provides the operating power only to the secondary power bus 364 of 
camera 110. 

FIGS. 4 and 5 are diagrams depicting the preferred hardware components of the 
20 camera's 110 user interface 408. FIG. 4 is back view of the camera 110 showing the 
LCD screen 402, a four-way navigation control button 409, an overlay button 413, a 
menu button 414, and a set of programmable soft keys 41 6. FIG. 5 is a top view of the 
camera 110 showing a shutter button 418, and a mode dial 420. The camera may 
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optionally include status LCD 406, status LCD scroll and select buttons 422 and 424, a 
sound record button 426, and zoom-in, zoom-out buttons 428a and 428b. 

The digital camera of the present invention is controlled by graphical-user- 
interface (GUI) based operating system (OS), which is in contrast to conventional digital 
5 cameras that are controlled by proprietary hardware architectures. In the preferred 
embodiment of the present invention, the OS provides the digital camera with several 
different operating modes for supporting various camera functions. Although the digital 
camera may include several different operating modes, the modes relevant to this 
description are capture mode, and play mode. 

10 In capture mode, the camera 100 supports the actions of preparing to capture an 

image, and capturing an image through the use of either the LCD screen 402 or the 
status LCD 406. In play mode, the camera 110 supports the actions of displaying full- 
sized views of captured images, and play-backing various media types associated with 
the images, such as sound. The user may switch between the various modes, using 

15 the mode dial 420. When the camera is placed into a particular mode, that mode's 
default screen appears in the LCD screen 402 in which a set of mode-specific items, 
such as images, icons, and text, are displayed. 

The present invention provides a method and system for controlling user 
interaction in a digital imaging device using dynamic overlay bars. According to the 

20 present invention, the dynamic overlay bars are used to provide the user with both 
status information and interactive instructions. The interactive instructions are 
automatically updated in response to normal camera operations to guide the user 
through predefined operations of the camera, thus making the device extremely easy to 
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use. In addition, the manner in which the dynamic overlay bars are displayed reduces 

viewing interference with the currently displayed object- 
Referring now to FIG. 6, a flow chart is shown illustrating the process of 

controlling user interaction in a digital imaging device using dynamic overlay bars in 
5 accordance with the present invention. The process begins by displaying an image on 

the LCD screen 402 along with at least one overlay bar that provides a dynamic prompt 

area in a way that minimizes viewing interference with the displayed image in step 450. 
In a preferred embodiment, viewing interference is minimized by positioning the 

overlay bar along an edge of the LCD screen 402 and by displaying the background of 
10 the bar translucently so that the user may see the image through the overlay bar. The 

overlay bar may also be displayed with a solid color background, but this is less 

desirable since the bar would overwrite that portion of the image. 

In response to the camera being placed into one of the operating modes, the 

overlay bar displays mode-specific information for the user in step 452. In a preferred 
15 embodiment, the mode-specific information displayed on the overlay bar includes a 

combination of static status information, dynamically updated soft key labels, and 

interactive instructions pertaining to the particular mode, as described further below. 

After the mode-specific information is displayed, the mode-specific information is then 

dynamically updated during the operation of the camera to guide the user through a 
20 mode-specific function in step 454. 

To more particularly describe the present invention, refer to FIGS. 7A and 7B 

illustrating the use of dynamic overlay bars on the LCD screen 402 during two different 

operating modes of the digital camera 110. As shown, in a preferred embodiment of 



P135 



-12- 



the present invention, two overlay bars 430 and 432 are simultaneously displayed on 
the LCD screen 402, rather than one, to strike a balance between the amount of 
information provided to the user and the amount of screen area consumed by text 
and/or graphics. 

5 Overlay bar 430 may be used primarily to display status information and 

interactive instructions, while overlay bar 432 may be used primarily to display soft key 
labels 410 corresponding to soft keys 412. Both overlay bars 430 and 432 may be 
turned-off in each of the camera operating modes by pressing the overlay "on/off 
button 413 so that users can have an unobstructed view of images if they so choose 

10 (off), or extra help in operating the camera (on). 

Referring to FIG. 7A, the display of the overlay bars 430 and 432 on the LCD 
screen 402 during capture mode is shown. In capture mode, the camera 110 supports 
the actions of preparing to capture an image, and capturing an image through the use 
of either the LCD screen 402 alone or with the aid of an optional optical viewfinder (not 

15 shown). 

Overlay bar 430 is updated with capture status information during capture mode, 
which may include a graphic memory gauge, and text indicating the state of the camera 
(Ready), for example. The memory gauge provides the user with a constant overview of 
camera memory usage in terms of disk space, and may also show working memory 
20 usage. In a preferred embodiment, the memory bar displays disk space usage as 
segments filling-up, and displays working memory usage as the bar below those 
segments, which is constantly updated to reflect current memory status. When the 
working memory buffers are empty, the bottom part of the bar would be clear. When 
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there is the equivalent of storage for only a few pictures left, the storage gauge may 
flash and the overlay bar 430 may be updated with a message, such as "Storage 
Almost Full". If a user tries to take a picture without adequate storage, then the overlay 
bar 430 may be updated to reflect this status by displaying the message "Inadequate 
5 Storage," along with an optional sound from the camera. 

The overlay bar 430 may also be updated to reflect other types of capture status 
information and may be expanded into additional lines if needed. The additional 
capture status information could include the following: 1) Low Battery Indication - when 
main batteries run low, a battery icon may replace the storage gauge and a overlay bar 
5 10 430 may be updated to flash "Battery Low"; 2) Shake Warning Indication - when light 
tf t level is too low for recommended hand held operation and user has disabled the strobe 
iT system "Shake Warning" may be displayed in the overlay bar 430; and 3) No Focus 
ftj Indication - when the focus system cannot adequately focus the camera lens, a "No 
Q Focus" may be displayed in the overlay bar 430. 

fy 15 Referring now to FIG. 7B, the display of the overlay bars 430 and 432 on the 

|f LCD screen 402 during play mode is shown. In a preferred embodiment, the play 
screen layout displays one full-sized image at a time and the user may chronologically 
scroll through the full-sized images in the LCD screen 402 using the left/right buttons on 
four-way navigation control button 409. Users can also play back various media types, 
20 such as time-lapse, bursts and slide show images according to either default or user 
defined play back rates. 

In the play mode, overlay bar 430 displays status information relating to the 
current image being displayed, such as the image name/number, and the date and time 
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of capture. The status information may also include graphical icons indicating what 
category of images the image belongs to and the image type. 

Referring to both FIGS. 7 A and 7B, besides displaying status information, the 
second use of the dynamic overly bars of the present invention is to display soft key 
5 labels 410 for soft keys 412. As described in U.S. Patent Application No. 08/939,993 
filed on September 26, 1997, entitled "A Method And System For Manipulating Images 
Stored In A Digital Imaging Device," assigned to the present assignee and hereby 
incorporated by reference, soft keys 412a, 412b, and 412c of the user interface 400 are 
programmable, i.e., they may be assigned predefined functions. The function currently 
% 10 assigned to a respective soft key 412 is indicated by the soft key labels 410a, 410b, 
J and 410c displayed in overlay bar 432. After a soft key label 410 has been displayed, 
fi the user may then press the corresponding soft key 412 to have the function indicated 
fl : by its label applied to the current image. 

p Referring to FIG. 7B for example, the function assigned to the soft key 412b in 

;^ 15 during play mode is a "Zoom" function, which allows a user to zoom in and out of a 
2; displayed image. When the user zooms-in on an image by pressing the soft key 412b, 
the "Zoom" soft key label 410b is changed to "Zoom-out". While an image is zoomed, 
the user may pan around the image using the four-way control button 406. 

The functions assigned to the soft keys 412, and thus the soft key labels 410, 
20 are changed in response to several different factors. The soft keys 412 may change 
automatically either in response to user actions, or based on predetermined conditions 
existing in the camera, such as the current operating mode, the image type, and so on. 
The soft keys 412 may also be changed manually by the user by pressing the menu 
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button 415. Providing programmable soft keys 412 increases the number of functions 
that may be performed by the camera, while both minimizing the number of buttons 
required on the user interface, and reducing the need to access hierarchical menus. 

As stated above, in addition to displaying status information and soft key labels, 
5 the dynamic overlay bars of the present invention may also be used to display 
interactive instructions to the user to guide user through camera functions. Basic types 
of camera functions include reviewing captured images, deleting images, annotating 
images with sound, and capturing groups of related images. With conventional 
cameras, the user would have to memorize complicated key sequences in order to 

1 0 perform these functions. 

The present invention, in contrast, uses the dynamic overlay bars to display 
interactive instructions that guide the user through operations such as adding sound to 
an image, deleting images and/or sound, and capturing groups of related images. 
As described in U.S. Patent Application No. 08/939,993, for example, after the user has 

15 captured an image and the image is displayed for review, the overlay bar 432 
automatically reminds the user that he or she has the option to delete the image. That 
is, one of the soft key labels 410 is changed to "Delete" and the user may then delete 
image by pressing the corresponding "Delete" soft key 412. 

Referring now to FIGS. 8A through 8C, diagrams illustrating how the overlay bars 

20 may be used to guide the user through a recording of a sound annotation are shown. 
The user may initiate the sound annotation function by pressing the record button 426 
(see FIG. 5) while an image is displayed. In response, a record indication, such as a 
microphone icon, is automatically displayed in overlay bar 430 along with a display of 
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the duration of the recording, as shown in FIG. 8A. After the sound annotation is 
recorded, the soft key labels 410 may be updated to display three options "Play", 
"Delete", and "Save"; where "Play" plays back the recorded sound, "Delete" deletes the 
recorded sound, and "Save" saves the recorded sound. 
5 If the user is reviewing images in play or review modes, it is possible that the 

displayed image will have a sound annotation attached. Should the user presses the 
"Delete" soft key 412, it is unclear what operation the user wishes to perform: delete the 
image, delete only the sound, or delete both. Indeed, an inexperienced user may not 
even consider all three of these possibilities before pressing the "Delete" button. 

10 Therefore, to guide the user through this operation, the dynamic overlay bars 430 and 
432 are updated to prompt the user whether the image or the sound annotation is to be 
deleted, as shown in FIG. 8B. The user may then indicate which is to be deleted by 
pressing the corresponding soft key 412. 

While reviewing images, it is also possible that the user may press the record 

15 button 426. If the current image already includes a sound annotation, then it is unclear 
whether the user wishes to record a new sound annotation over the old one, or whether 
the user is unaware of the existing sound annotation. Therefore, to make sure the user 
doesn't inadvertently overwrite the existing sound, the overlay bar 430 is automatically 
updated to inform the user that sound will not be recorded until the user deletes the 

20 existing sound, as shown in FIG. 8C. In addition, if the user doesn't recall the contents 
of the previous sound annotation, the user may listen to it before deleting it by pressing 
"Play", or the user may cancel the record operation altogether by pressing "Exit". Thus, 
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according to the present invention the user is enabled to perform complex tasks in the 

camera without fumbling through a set of hierarchical menus. 

Another use of displaying interactive instructions in the dynamic overlay bars 430 

and 432 in accordance with the present invention is to direct the user through image 
5 capture sequences. The purpose of directed image capture sequences is to customize 

the camera's image capture process for a specific application. More specifically, a 

directed image capture is a camera feature that provides the user with interactive 

instructions and feedback during capture mode to guide the user through a series of 

task-oriented image captures. 
^ 10 Upon initiation of a directed image capture sequence, interactive instructions are 

J?t displayed the dynamic overlay bars 430and 432 that prompt the user to perform 

specific operations (capture image or capture sound), and for prompting the user to 
fU enter specific input (name and date). Customized directed image captures can be 
3 tailored to specific professions, such as insurance claims adjusters and real estate 
M 15 agents, who would benefit from the use of a digital camera to capture groups of related 
S pictures. 

Referring now to FIGS. 9A and 9B, diagrams illustrating example directed image 
capture screens are shown. The example shown in FIG. 9A may pertain to an 
insurance-related directed image capture that prompts an insurance claims adjuster to 
20 take a series of pictures of a damaged vehicle, or it may pertain to a real estate 
application that guides a user through taking photos of a house for sale. 

In the insurance example, once the directed image capture has started, the user 
may be instructed to take various views of the damaged car. The user may also be 
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shown the number of the current image in that sequence, and the total number of 
images to be captured. 

After the views of the car are taken, the directed image capture may then prompt 
the user to enter specific information, such as the name of the image, as shown in FIG. 
5 9B. The user may then enter text by choosing letters using the four-way control button 
409. For insurance purposes, the directed image capture may also request the user to 
input the owner's name, license plate number, claim number, and so on. The sequence 
of images and corresponding information may then be downloaded from the camera or 
to a host computer for automated database storage or web page generation, 
y 10 In one embodiment of the present invention, one or more directed image capture 

~Tfi sequences may be provided in the camera as built-in functions, especially if the camera 
§2 is tailored for specific industries. 

ry However, in a second aspect of the present invention, the camera is made more 

Q flexible by implementing the directed image capture sequences as a set of program 
ft 15 instructions that are externally loaded into the camera. Once loaded in the camera 110, 

the instructions are then preferably executed by the GUI-based system software 

running on CPU 344. 

FIG. 10 is a block diagram illustrating the contents of ROM 350 where the 

software is stored, and DRAM 346 where the software is executed. The software 600 
20 may include a control application 602, a toolbox 604, drivers 612, a kernel 614, and a 

startup/ configuration module 616. The control application 602 is the main program that 

controls high-level functions of the digital camera and is responsible for interfacing with 

functions in the toolbox 604. 
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Toolbox 604 comprises selected function modules that control how the digital 
camera captures and manipulates images. The modules may include image 
processors 606, a camera control shell 608, and a script interpreter 610. Image 
processors 606 are programs for enhancing (e.g., adjusting the contrast, sharpening, 
5 converting the image to gray-scale, etc.) the digital image received from imaging device 
114. Camera control shell 608 receives and processes data structures for controlling 
camera functions. Script interpreter 610 translates and executes script statements, 
which are used to provide the directed image capture sequences and other camera 110 
features, as explained below, 
y 10 Drivers 612 comprise program instructions for controlling various camera 110 

r$ hardware components, such as motor 234 (FIG. 2) and a flash (not shown). Kernel 614 
I? comprises program instructions providing basic underlying camera operating system 
111 services including synchronization routines, task creation, activation and deactivation 
Q routines, resource management routines, etc. Startup/configuration 616 comprises 
fM 15 program instructions for providing initial camera 110 start-up routines such as the 
Jf system boot routine and system diagnostics 

When the camera 110 is first turned on and booted up, the startup/configuration 
616 module begins to execute and loads the drivers 612, the kernel 614, the control 
application 602, and system files containing configuration information into DRAM 346. 
20 Thereafter, operation of the camera is passed to the control application 602. In an 
alternative embodiment, the software 600 may executed out of ROM 350 in order to 
reduce the size of DRAM 346. 
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The directed image capture sequence 618 may be loaded into the digital camera 
110 from the removable memory 354 (FIG. 3), a host computer, or a network, and 
stored in DRAM 346 to run in place of the control application 602. In a preferred 
embodiment, the directed image capture sequence 618 is implemented using a script, 
5 which is a program written with text-based commands that may be easily written by the 
user. As used herein, a script may be written in any intrepreted language, such as 
Basic and Lisp, for example. 

Once loaded into the camera, the script may be selected by the user from a 
menu where it is displayed for selection, and is thereafter executed by the control 
5 10 application 602 by passing the script to the script interpreter 610, The script interpreter 
•** £ 610 then translates and executes the script instructions comprising the directed image 
12 capture sequence 618 one-by-one. 

ry In an alternative embodiment, a directed image capture sequence 618 may be 

O implemented as a traditional application program, rather than a script. However, an 
fy 15 application program is typically written by a software developer in a traditional computer 
!^ language, such as C++, compiled, and stored in machine language, which is a more 

complicated process than adding new functions to the camera via a text-based 

interpreted script. 

FIG. 1 1 is a flow chart illustrating an exemplary process of installing and running 
20 a script-based directed image capture in a preferred embodiment of the present 
invention. The process begins by inserting the removable memory 354 in step 700. 
When the removable memory 354 is installed, the removable memory 354 is mounted 
by the operating system 600 in step 702. Thereafter, the operating system searches for 
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system files on the removable memory 354, which alert the digital camera 1 1 0 to the 
presence of an external program, in step 704. 

Any system files found on the removable memory 354 and corresponding 
directed image capture sequences 618 are then installed and made available to the 
5 user for selection via menu choices that appear on the LCD screen 402 in step 706. In 
a preferred embodiment, steps 704 and 706 are implemented as a hot-mount process 
when the removable memory 354 is inserted into the camera 1 10, as described in U.S. 

patent application No. , entitled "Method And System For Dynamically 

Updating Software Functions In A Digital Capture Device (P149)," filed on , 

10 1998, which is assigned to assignee of the present application and herein incorporated 
by reference. 

Once the list of available directed image capture sequences 618 are displayed, 
the user selects one of the directed image capture sequences 618 to run in step 708. 
In a preferred embodiment, the list showing the available directed image capture 

15 sequences may be categorized in menus for easier selection. For example, assume a 
real estate agent has three different scripts for capturing images of different types of 
properties. The agent may name or create categories for the directed image capture 
sequences called "commercial", "industrial", and "residential", for instance. Selecting 
the residential category, for example, will cause a list of directed image captures to be 

20 displayed that are designed to capture pictures of different types of residential 
properties, such as one, two, and three bedroom homes. The user may then select a 
desired script depending on the particular house to be shot. 
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In one preferred embodiment, the directed image capture selections displayed in 
the menus may be erased from the camera by rebooting the camera, or by removing 
the removable memory 354 from the camera 110. 

After the user selects one of the directed image capture sequences 618 to run, 
5 the script interpreter 610 begins interpreting the directed image capture sequence 618 
in step 710, and control is passed from the control application 602 to the script. In step 
712, the script interpreter 610 fetches the first command comprising the directed image 
capture sequence 618. 

It is then determined whether the fetched command is a script "WaitForShutter" 
y 10 command in step 714. This command causes control of the camera 1 10 to pass back 
h to the control application 602 until the user presses the shutter button 418 to capture an 
image. The "WaitForShutter* command is preferably called with a quoted string 
fy parameter that is used in the dynamic overly bar 430 as the prompt to the user 
D requesting an image capture (e.g., "Take photo of kitchen"). 

ft! 15 If the command is a "WaitForShutter" command in step 714, then control is 

|J returned to the script after the user presses the shutter button 418 in step 716 to 

capture an image. If the fetched command is not a "WaitForShutter" command in step 
714, then the script interpreter 610 interprets and executes the command in step 718. 
After the user presses the shutter button 418 or after a script command has been 
20 executed, it is determined if the end of the script has been reached in step 720. If not, 
then the next command is fetched in step 712, and the process continues until the end 
of the script is reached, at which point control is returned to the control application 602 
in step 722. 
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Besides the "WaitForShutter" command, scripts may include two other 
categories of script commands. One category of commands pertain to camera settings, 
controls and other camera parameters specific to the subject and/or scene being 
captured, (ie: White Balance Modes, Exposure Modes, and Focus Modes). This 
5 category of commands enable users to input "Hints" optimizing the camera's photo 
systems for specific photographic conditions. 

The other category of commands may pertain to file system operations and 
image tagging functions specific to the way in which image data is stored in memory, 
(ie: Guided Capture, Prompted Text/Audio Annotation, and Automated Image 
10 Grouping/Cataloging/Indexing.) This category of commands is particularly useful when 
used in conjunction with desktop computer applications where the hosting application is 
coordinated to take advantage of the preformatted media organization and tag 
information. For example, while a directed image capture sequence guides the user 
though a series of steps to create an image grouping, the script commands comprising 
15 the sequence generate appropriate tags and data structures to group the images and 
text captured during the sequence. 

No matter whether the dynamic overlay bars of the present invention are used to 
display status information, soft key labels, or interactive instructions, as described 
herein, one important component affecting the user's experience is the method used to 
20 display the overlay bars on the image. 

One approach would be the follow prior art techniques for displaying text (e.g. 
image name) over an image. This approach typically includes the following steps: 1) 
fetching the image to be displayed, which is typically stored in JPEG format, 2) 



P135 



-24- 



decompressing and resizing the image, 3) displaying the decompressed image block- 
by-block, and then after the image is fully displayed, 4) writing the text on top of the 
image. 

The problem with this method is that is visually unappealing to the user, and it 
5 reduces the performance of camera when the user turns-off the text display while 
viewing the image. The reason the method reduces camera performance is the 
following. When text or graphics are displayed over the image, they obscure a portion 
of the image. And when the text is turned-off, the obscured portions of the image must 
be displayed so that the original image is seen without the text. In order to do this, 

10 however, the entire JPEG image must be fetched and decompressed again so that the 
obscured portions of the image can be displayed on the LCD, which can be a time 
consuming operation. 

A third aspect of the present invention overcomes these disadvantages by 
providing an improved method and system for displaying the overlay bars that not only 

15 enhances the visual effect associated with the overlay bars, but also eliminates the 
need to re-decompress the JPEG image data when the user turns-off the overlay bars, 
thereby increasing performance of the camera. 

According to this aspect of the present invention, the overlay bars are displayed 
first, followed by the image, wherein the image is made to appear as though it is sliding 

20 underneath the overlay bars as it is being displayed. The image appears as though is it 
is sliding underneath the overlay bars because the image is displayed on the LCD 
screen 402 line-by-line or block-by-block (as used herein, a block may include 
anywhere from one line to sixteen lines of image data). As the display of the image 
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progresses from the top of the screen 402, the image therefore appears to be displayed 
behind the overlay bars 430 and 432 which are already present on the LCD screen 402, 
The overlay bars 430 and 432 are also provided with a translucent background 
so that so that the overlay bars 430 and 432 themselves do not obscure the image, but 
5 the text is easily distinguishable from the colors of the displayed image. The result is 
that after the image has been displayed, the overlay bars appear as a separate layer 
over the image. Further, the portions of the original image that intersect with the 
overlay bars 430 and 432 are saved, so that when the user turns-off the overlay bars 
430 and 432, only these portions of the image are redisplayed to restore the image. 

10 Thus this aspect of the present invention eliminates the need to re-decompress and 
display the entire image again, thereby increasing system performance. 

Where typically, specialized hardware would be required to achieve the above- 
described effects, the present invention accomplishes the task through software and 
the manipulation of several memory buffers, as shown in FIG. 12A. 

15 FIG. 12A is a diagram illustrating a buffer organization for displaying overlay 

bars, which in a preferred embodiment, resides in DRAM 346. The buffer organization 
includes an overlay bar buffer 540, a backstore buffer 542, and a display buffer 544. 
According to the present invention, the overlay bar buffer 540 is used to store the 
graphics data (graphics and text) that will be displayed in the overlay bars 430 and 432. 

20 In a preferred embodiment the overlay bar buffer 540 is divided into a top and bottom 
portion, which store twenty lines of data each that correspond to the top and bottom 
overlay bar 430 and 432, respectively. 
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The backstore buffer 542 is used to store original image data corresponding to 
the area of the LCD screen 402 where the overlay bars 430 and 432 will be displayed. 
The backstore buffer 542 is also divided into a top and bottom portion that are the same 
size as the top and bottom portions of the overlay bar buffer 540. 
5 As is typical in most rendering systems, the display buffer 544 is used to store 

the actual data that is to be displayed on the LCD. The data in the display buffer is 
accessed by LCD controller 390 (FIG. 3) and displayed on the LCD. 

FIG. 12B is a flow chart illustrating the process of displaying overlay bars on the 
LCD in accordance with the present invention. The first step in the process is to 

10 preferably receive an input line of decompressed image data from an image processing 
system in step 800. The process may also be modified to receive an input block of 
decompressed image data. In a preferred embodiment, the image processing system 
for providing the input data may include an image decompressor for decompressing the 
image data, and a resizer for resizing the lines of image data to fit the size of LCD 

15 screen 402. 

Next, it is determined whether the overlay bars 430 and 432 are turned-on or off 
in step 802. If the overlay bars are turned-off, then the line of image data is copied 
directly to the display buffer 544 in step 804 for display on the LCD screen 402 and the 
process continues. If the overlay bars remain off for the duration of the time it takes to 
20 display the image line-by-line or block-by-block, then the entire image is displayed on 
the LCD screen 402 using only the display buffer 544. 

If the overlay bars are turned-on in step 802, then it is determined whether the 
line of data will be displayed within the area of the LCD screen 402 that is occupied by 
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an overlay bar in step 806. If the line is within an overlay bar, the line is copied into the 
backstore buffer 542 in step 808. The purpose of copying the line to the backstore 
buffer 542 is to save the portion of the image that will be displayed underneath the 
overlay bars 430 and 432. 
5 After the current line of image data is copied into the backstore buffer 542, the 

corresponding line stored in the overlay bar buffer 540 is merged with the current line in 
the backstore buffer 542 in step 810. The purpose of merging the two lines is to display 
the background of the overlay bars 430 and 432 translucently over the image on the 
LCD screen 402. This is done by halving the luminance value of each pixel of the 

10 image data from the backstore buffer 542 that falls within the bounds of an overlay bar 
430 or 432, and overwriting each pixel in the line of image data that falls under a pixel 
of text or graphic data from the overlay bar buffer 540. Halving the luminance value of 
the image data causes the colors of the image that overlap an overlay bar 430 or 432 to 
be half as bright, thus giving the overlay bar 430 or 432 a translucent appearance and 

1 5 allowing the user to see the image through the overlay bar 430 or 432, as shown in FIG. 
7B. In an alternative embodiment, the translucency of the overlay bars 430 and 432 is 
provided by increasing, rather than decreasing, the luminance value of each image 
pixel falling within the area of an overlay bar. In this case, the text displayed in the 
overlay bars 430 and 432 is displayed using a dark color. 

20 As the line from the overlay bar buffer 540 is merged with the line from the 

backstore buffer 542, the resulting merged line is written into the display buffer 544 for 
display in step 812. If the current line is the last line of image data in step 814, then the 
process ends. Otherwise the next line of image data is received in step 800 and the 
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process continues. In an alternate embodiment of the present invention, the 
determination of whether the overlay bars 802 are on/off in step 802 may be performed 
after copying the input line to the backstore buffer 542 in step 8. In this embodiment, 
the input line is copied into the backstore buffer 542 even when the overlay bars 430 
5 and 432 are off. 

In a preferred embodiment of present invention, the software 600 controlling the 
digital camera 110 is implemented as event driven software, which responds to input 
from the user (select menu, press button, etc.) or other applications at unregulated 
times. When, for example, the user first switches to play mode and/or selects a new 
"*jf 10 image to display, the first steps that are performed in the process are to blank the LCD 
ffi screen 402, fill the overlay bar buffer 540 with relevant mode-specific information, and 
i2 then contents of the overlay bar buffer 540 and the backstore buffer 542 are merged 
fy and written to the display buffer 544. In this case, the backstore buffer 542 may contain 
G black or white pixel values to provide the blank screen. Thereafter, the process 
W 15 proceed as described in FIG. 13. 

If the user turns-off the overlay bars 430 and 432 while an image is displayed, 
then the process is interrupted and software 600 copies the entire contents of the 
backstore buffer 542, which contains the original image data, to the display buffer 544 
for display. This causes the overlay bars to disappear from the LCD screen 402 and 
20 restores the original image without having to re-decompress and display the entire 
image over again. 

If the user then turns-on the overlay bars 430 and 432, the software 600 merges 
the contents of the overlay bar buffer 540 and the backstore buffer 542 to provide the 
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translucent bars and text over the image, and then copies the result to the display 
buffer 544 for display. This may be done by executing step 812 and 814 for each line 
of the data in the buffers 540 and 542. 

Also, when the overlay bars 430 and 432 are on, if the overlay bars 430 and 432 
5 are updated by the control application 602 due to a change in status or instructions, the 
contents of the overlay bar buffer 540 and the backstore buffer 542 are remerged and 
written into the display buffer 544 for display. 

A method and system for controlling user interaction in a digital imaging device 
using dynamic overlay bars has been disclosed, which enables a user to apply camera 

10 functions and features to images with minimum effort, while allowing for easy viewing of 
the image. In addition, the present invention displays dynamic interactive instructions to 
the user in the form of directed image capture to guide the user through complex task, 
without the need for help screens or for the user to remember complicated key 
sequences. Finally, the present invention provides a method for displaying overlay bars 

15 that eliminates the need to re-decompress and display the image when the user turns- 
off the overlay bars, which increases the responsiveness of the camera. 

Although the present invention has been described in accordance with the 
embodiments shown, one of ordinary skill in the art will readily recognize that there 
could be variations to the embodiments and those variations would be within the spirit 

20 and scope of the present invention. For example, the functions assigned to the soft 
keys, the number of soft keys, and the placement of the soft keys and labels in and 
around the display may vary. The method and system may also be implemented in digital 
imaging devices having only two modes, but that have multiple navigation screens within 
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the "play mode" Accordingly, many modifications may be made by one of ordinary skill 
in the art without departing from the spirit and scope of the appended claims. 



CLAIMS 

5 What is claimed is: 

A method for controlling user interaction in a digital imaging device having 
using dynamic overlay bars, the method comprising the steps of: 

providing the digital imaging device with at least two operating modes, 
each of the operating modes having at least one mode-specific operation 
that can be performed on images; 

in response to operating in either of the at least two operating modes, 
displaying at least one overlay bar on the display; and 
dynamically updating the at least one overlay bar with both status 
information and interactive instructions, wherein the interactive 
instructions guide the user through at least one the mode-specific 
operation. 

2 The method of claim 1 wherein step c) further includes the step of 
20 providing the interactive instructions for guiding the user through a directed image 
capture operation. 
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3 The method of claim 2 wherein step c) further includes the step of 
providing the interactive instructions for guiding the user through a sound annotation 
operation. 

5 4 The method of claim 2 wherein step c) further includes the step of 

dynamically updating the first overlay bar using an application program. 

5 The method of claim 4 wherein the digital imaging device includes a 
display having a fist and a second horizontal sides, step b) further including the step of 
y 1 0 displaying the at least one overlay bar adjacent to a first horizontal side of the display. 

12 6 The method of claim further 5 including the step of displaying a second 

ry overlay bar adjacent to a second horizontal side of the display. 

W 15 7 The method of claim further 1 including the step of displaying the at least 

Jf- one overlay bar translucently on the display. 

8 A method for controlling user interaction in a digital imaging device using 
dynamic overlay bars, the digital imaging device including a display, a first function key, 
20 and at least two operating modes for capturing, storing, and reviewing the images, the 
method comprising the steps of: 

a) displaying mode-specific information on a first overlay bar that is 
displayed translucently over a current image in the display; 
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b) displaying a mode-specific function label corresponding to the first 
function key in a second overlay bar, wherein the mode-specific function 
label indicates which one of a plurality of predefined functions is assigned 
to the first function key; and 

c) dynamically changing the mode-specific information and the mode- 
specific function label during operation of the digital imaging device to 
automatically guide a user through operation of the digital imaging device. 

9 The method of claim 8 further including the step of: 

e) automatically changing the predefined function assigned to the first 
function key and the mode-specific function label based on conditions existing in the 
digital imaging device. 

1 0 The method of claim 9 further including the step of: 

f) manually changing the predefined function assigned to the first function 
key by pressing a menu key. 

1 1 The method of claim 1 0 wherein step a) further includes the step of: 
a1) providing the current image as a live view image during capture mode. 

12 The method of claim 1 1 wherein step a) further includes the step of: 
a2) providing the current image as a previously stored image. 
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13 The method of claim 12 wherein step b) further includes the step of: 
b1) displaying the mode-specific function label as a combination of an icon 
and text. 



5 14 A digital imaging device comprising: 

an imaging device for capturing image data; 

a memory coupled to the imaging device for storing the image data as captured 
images; 

a display for displaying a captured image; 
1 0 a processor coupled to the imaging device and to the memory for controlling 

operation of the digital imaging device and for operating the image device in at least two 
operating modes; and 

at least one overlay bar displayed on the display, the at least one overlay bar 
including a translucent background and a combination of mode-specific status 
15 information and interactive instructions, wherein the interactive instructions 
automatically guide a user through a mode-specific operation. 

15 A digital imaging device as in claim 14 further including a set of soft keys, 
wherein the at least one overlay bar further displays soft key labels to indicate which 
20 one of a plurality of functions is assigned to the corresponding soft key, wherein in 
response to a user pressing a first one of the soft keys, the processor applies the 
function assigned to the first one of the soft keys to the first captured image. 
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16 A digital imaging device as in claim 15 wherein the at least one overlay 
bar is updated automatically based on conditions existing in the digital imaging device 

17 A digital imaging device as in claim 16 wherein the captured image is a 
live image displayed during a capture mode. 

18 A digital imaging device as in claim 17 wherein the captured image is 
previously captured and stored image. 
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ABSTRACT 

A method and system for controlling user interaction in a digital imaging device 
having a display using dynamic overlay bars. The digital imaging device includes at 
least two operating modes, where each of the operating modes has at least one mode- 
5 specific operation that can be performed on images. In response to operating in both 
of the operating modes, the digital imaging device displays a first overlay bar on the 
display that is dynamically updated with status information and interactive instructions 
that guide the user through the mode-specific operations. 
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□ 

(Number) (Country) (Day/Month/Year Filed) 

□ 



(Number) (Country) (Day/Month/Year Filed) 

□ 

(Number) (Country) (Day/Month/Year Filed) 
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I hereby claim the benefit under 35 U.S.C. Section 119(e) of any United States provisional 
application(s) listed below: 



(Application Serial No.) 


(Filing Date) 


(Application Serial No.) 


(Filing Date) 



(Application Serial No.) 



(Filing Date) 



I hereby claim the benefit under 35 U. S. C. Section 120 of any United States application(s), or 
Section 365(c) of any PCT International application designating the United States, listed below and, 
insofar as the subject matter of each of the claims of this application is not disclosed in the prior 
United States or PCT International application in the manner provided by the first paragraph of 35 
U.S.C. Section 1 12, I acknowledge the duty to disclose to the United States Patent and Trademark 
Office all information known to me to be material to patentability as defined in Title 37, C. F. R., 
Section 1.56 which became available between the filing date of the prior application and the national 
or PCT International filing date of this application: 



(Application Serial No.) 


(Filing Date) 


(Status) 

(patented, pending, abandoned) 


(Application Serial No.) 


(Filing Date) 


(Status) 

(patented, pending, abandoned) 


(Application Serial No.) 


(Filing Date) 


(Status) 

(patented, pending, abandoned) 



I hereby declare that all statements made herein of my own knowledge are true and that all 
statements made on information and belief are believed to be true; and further that these statements 
were made with the knowledge that willful false statements and the like so made are punishable by 
fine or imprisonment, or both, under Section 1001 of Title 18 of the United States Code and that such 
willful false statements may jeopardize the validity of the application or any patent issued thereon. 
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POWER OF ATTORNEY: As a named inventor, I hereby appoint the following attorney(s) and/or 
agent(s) to prosecute this application and transact all business in the Patent and Trademark Office 
connected therewith, (list name and registration number) 
Stephen G. Sullivan, Reg. No. 38,329 



Send Correspondence to: Ste P hen G - Sullivan 

152 N. Third Street, #800 
San Jose, CA 95112 



Direct Telephone Calls to: (name and telephone number) 

Stephen G. Sullivan (408) 795-4925 



Full name of sole or first inventor 
Eric C. Anderson 

/ , Date 
Residence 

931 Brentwood Drive, San Jose, CA 95129 
Citizenship 

USA 

Post Office Address 

931 Brentwood Drive, San Jose, CA 95129 



Sole or first inventor's 




Full name of second inventor, if any 



Second inventor's signature Date 



Residence 



Citizenship 



Post Office Address 
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